Methods and apparatus for use in speech recognition systems for identifying unknown words and for adding previously unknown words to vocabularies and grammars of speech recognition systems

ABSTRACT

The present invention concerns methods and apparatus for identifying and assigning meaning to words not recognized by a vocabulary or grammar of a speech recognition system. In an embodiment of the invention, the word may be in an acoustic vocabulary of the speech recognition system, but may be unrecognized by an embedded grammar of a language model of the speech recognition system. In another embodiment of the invention, the word may not be recognized by any vocabulary associated with the speech recognition system. In embodiments of the invention, at least one hypothesis is generated for an utterance not recognized by the speech recognition system. If the at least one hypothesis meets at least one predetermined criterion, a word or more corresponding to the at least one hypothesis is added to the vocabulary of the speech recognition system. In other embodiments of the invention, before adding the word to the vocabulary of the speech recognition system, the at least one hypothesis may be presented to the user of the speech recognition system to determine if that is what the used intended when the user spoke.

TECHNICAL FIELD

The invention concerns methods and apparatus for use in speechrecognition systems and more particularly concerns methods and apparatusfor identifying and assigning meaning to new words and utterances. Thenew words and utterances may be known beforehand, but used in a new wayunknown to an embedded grammar of a language model incorporated in aspeech recognition system, or may be totally unknown beforehand from anyperspective to a speech recognition system.

BACKGROUND

Speech recognition systems are finding increasing use, particularly invoice-controlled user interfaces. Voice-controlled user interfaces arefamiliar to anyone who performs banking and credit card transactions bytelephone. In the past, telephonic banking and credit card servicetransactions were performed either through interaction with a humanagent or by using a keypad of a telephone; now, with increasingfrequency telephonic banking and credit card service transactions may beperformed using voice commands.

Voice-activated user interfaces are also finding increasing use inportable electronic devices like cellular telephones and personaldigital assistants (“PDAs”) with telephonic capabilities. For example,in cellular telephones with voice-activated user interface capability, auser can enter a voice command “Call Bob Smith” in order to initiate atelephone call to a target person (“Bob Smith”). This eliminates theneed for the user to enter a telephone number, or to access a contactlist containing the telephone number, thereby saving keystrokes. Theelimination of keystrokes often enables hands-free modes of operation,which is particularly advantageous when the telephone call is initiatedby someone operating an automobile. There is increasing pressure torestrict the operation of cellular telephones by drivers of automobiles,particularly cellular telephones that require hand operation.

Thus, the ability to initiate an operation (e.g., a telephone call) byissuing a voice command to a voice-controlled user interface isparticularly advantageous because it saves time and effort previouslyexpended by entering commands using keys or other hand-operated inputdevices. This advantage ends, though, as soon as a user enters a commandnot recognized by a speech recognition system associated with avoice-controlled user interface. In such circumstances, a user is oftenthrust back to old, more tedious modes of operation where a command hasto be entered using a combination of keystrokes.

In such situations, where a cellular telephone user is seeking toinitiate a telephone call, the user would either have to enter thetelephone number directly, or add it to a contact list. Since users ofproductivity-enhancement devices like cellular telephones and PDAs valuethe ability of these devices to “grow” with the user by, for example,being able to record and save an extensive and ever-expanding contactlist, the fact that this ability may only be partially implemented (ifat all) through voice commands is viewed as a particular limitation ofvoice-activated user interface systems incorporated in such devices. Ifa user has an extensive contact list, the user might not even initiate atelephone call using the voice command feature, because the user mightforget whether the person to be called is even in the contact list andthus capable of being recognized by a voice-activated user interfaceoperating in combination with the contact list.

A further problem is apparent in this description of the prior art. Inconventional speech recognition systems, the vocabularies and grammarsare fixed. Accordingly, when the user is thrust back upon akeystroke-mode of operation in order to enter new commands, the userwill have to enter the new commands with keystrokes every time the newcommands are to be performed, since the vocabularies and grammars arefixed. There is no benefit to the speech recognition system associatedwith the user giving meaning to a command unrecognized by the speechrecognition system using keystrokes, since the information entered usingkeystrokes does not modify the capabilities of the speech recognitionsystem.

Accordingly, those skilled in the art desire speech recognition systemswith the ability to “grow.” In particular, those skilled in the artdesire speech recognition systems with the ability to identify new wordspreviously unknown to the speech recognition system and to add them toone or more vocabularies and grammars associated with the speechrecognition system. In addition, those skilled in the art desire voiceactivated user interfaces with the ability to learn new commands.Further, when it is necessary to enter commands using keystrokes, thoseskilled in the art seek speech recognition systems that can bere-programmed though interaction with keys, keyboards, and other commandentry controls of an electronic device, so that the speech recognitionsystem benefits from the efforts expended in such activities.

SUMMARY OF THE PREFERRED EMBODIMENTS

The foregoing and other problems are overcome, and other advantages arerealized, in accordance with the following embodiments of the presentinvention.

A first embodiment of the present invention comprises a signal-bearingmedium tangibly embodying a program of machine-readable instructionsexecutable by a digital processing apparatus of a computer system toperform speech recognition operations, the speech recognition operationscomprising: detecting at least a target word known to an acousticvocabulary but unknown to an embedded grammar of a language model of thespeech recognition system; assigning a language model probability to thetarget word; calculating a sum of an acoustic and language modelconfidence score for the target word and words already included in theembedded grammar of the language model; and if the sum of the acousticand language model probability for the target word is greater than thesum of the acoustic and language model probability for the words alreadyincluded in the embedded grammar, adding the target word to the languagemodel.

A second embodiment of the present invention comprises a signal-bearingmedium tangibly embodying a program of machine-readable instructionsexecutable by a digital processing apparatus of a computer system toperform speech recognition operations, the speech recognition operationscomprising: detecting an utterance having a low acoustic score within anacoustic vocabulary of the speech recognition system indicating that theutterance may correspond to an out-of-vocabulary word; generating atleast one new word hypothesis comprised of at least one of a phone- orsyllable sequence using confidence scores derived from probabilitiescontained in a database of viable phone and syllable sequences; and ifthe at least one new word hypothesis meets a pre-determined criterion,adding a word corresponding to the at least one new word hypothesis tothe vocabulary of the speech recognition system.

A third embodiment of the present invention comprises a signal-bearingmedium tangibly embodying a program of machine-readable instructionsexecutable by a digital processing apparatus of a computer system toperform speech recognition operations in a speech recognition system,the speech recognition operations comprising: detecting an utterance notrecognized by at least a first one of an acoustic vocabulary, embeddedgrammar, and viable phone/syllable sequence library of the speechrecognition system; generating at least one hypothesis for theutterance, wherein the hypothesis is based on information derived from asecond one of an acoustic vocabulary, embedded grammar and viablephone/syllable sequence library of the speech recognition system;calculating a confidence score for the at least one hypothesis and formembers of the first one of the acoustic vocabulary, embedded grammarand viable phone/syllable sequence library of the speech recognitionsystem; comparing the confidence scores calculated for the at least onehypothesis and for members of the first one of the acoustic vocabulary,embedded grammar and viable phone/syllable sequence library of thespeech recognition system; and adding information to the first one of anacoustic vocabulary, embedded grammar and viable phone/syllable sequencecorresponding to the hypothesis if a pre-determined criterion based onthe comparison is met.

A fourth embodiment of the present invention comprises a speechrecognition system comprising: a speech input for receiving speech froma user of the speech recognition system; an open set comprised of atleast one open vocabulary and at least one open embedded grammarassociated with a language model implemented in the speech recognitionsystem; a hierarchical mapping system for identifying utterances notrecognized by at least one of the open vocabulary and open embeddedgrammar of the speech recognition system; for generating hypotheses forthe unrecognized utterances using confidence scores based at least inpart on one of viable phone/syllable sequence information, acousticvocabulary information and grammar information; and for addinginformation corresponding to the hypotheses to at least one of the openvocabulary and embedded grammar of the speech recognition system if apre-determined criterion is met; and a confidence score system forgenerating confidence scores for use by the hierarchical mapping system.

In conclusion, the foregoing summary of the alternate embodiments of thepresent invention is exemplary and non-limiting. For example, one ofordinary skill in the art will understand that one or more aspects orsteps from one alternate embodiment can be combined with one or moreaspects or steps from another alternate embodiment to create a newembodiment within the scope of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other aspects of these teachings are made more evidentin the following Detailed Description of the Preferred Embodiments, whenread in conjunction with the attached Drawing Figures, wherein:

FIG. 1 is a block diagram depicting a system embodying several aspectsof the present invention;

FIG. 2 is a block diagram depicting in greater detail the hierarchicalmapping system of FIG. 1;

FIG. 3 is a block diagram depicting a phone/syllable mapper made inaccordance with the present invention;

FIG. 4 is a block diagram depicting a user behavioral biometricsdetector made in accordance with the present invention; and

FIG. 5 is a flow chart depicting a method operating in accordance withthe present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

As introduction, an aspect of the present invention will be described toillustrate problems encountered in the prior art and how the presentinvention solves them. Embodiments of the present invention aregenerally operative in automated, electronic speech recognition systemsthat are used in electronic devices with speech input capability suchas, for example, telephones. The speech recognition systems typicallyoperate in such electronic devices as part of a voice-activated userinterface. Before the electronic device can take action in response to auser command, the speech recognition system has to parse the speechutterance comprising the command and assign meaning to the speechutterance. In prior art devices, users are required to operate withinrelatively narrow categories of vocabulary and grammar when interactingwith a speech recognition system, because conventional speechrecognition systems are fixed in capability. The speech recognitionsystems of the prior art have fixed vocabularies and grammars, meaningthat if a speech utterance is not in a speech recognition system'svocabulary and grammar, no action or possibly even an incorrect actionwill be taken by the voice-activated user interface. This occurs becausethe speech utterance is unknown to the speech recognition systemassociated with the voice activated user interface.

Accordingly, an aspect of the present invention provides a speechrecognition system with open vocabularies and grammars, allowing thespeech recognition system to be programmed with new words andgrammatical constructs (such as, for example, commands) throughinteraction with a user. As a result of these interactions, avoice-activated user interface with which the speech recognition systemis associated can be programmed to perform new actions. To illustratethe operation of an aspect of the invention an example will be provided.Assume a user is interacting with a voice-activated user interface thatis incorporated in a telephone, and speaks a command “Call Morita-san”.“Morita” is a Japanese surname, and “Morita-san” is a way one named“Morita” may be addressed in Japanese. The speech recognition system isprogrammed to recognize the command “Call ______”, and also isprogrammed to recognize certain names and telephone numbers that areused in combination with the “Call ______” command. However, in thisparticular example, the speech recognition system is initially notprogrammed to recognize the name “Morita-san”, nor has the userheretofore uttered the words “Morita-san” in combination with thecommand “Call ______”. Accordingly, in one aspect of the presentinvention, the speech recognition system generates a phonetic sequencehypothesis for “Morita-San” having a high degree of probability;presents the hypothesis to the user for confirmation, includingspelling; and after receiving confirmation (and possibly even a spellingcorrection) adds the word “Morita-San” to an embedded grammar associatedwith the “Call ______” command. In various embodiments of the invention,additional steps may be performed. For example, the user may associate aspecific telephone number with the word “Morita-san” as it is beingadded to the embedded grammar of the speech recognition system. Once“Morita-san” has been added to the embedded grammar and the telephonenumber has been associated with the new word “Morita-san”, the next timethe speech recognition system hears the command “Call Morita-san” itwill automatically call the telephone number associated with“Morita-san”.

In variants of this embodiment, confidence scores may be assigned usingadditional information besides, for example, phonetic or grammarinformation. Higher-level models based on semantic and contextinformation may be used in combination with phonetic and grammarinformation to identify unknown words using confidence scores. Forexample, regarding context, the speech recognition system may take intoconsideration what actions the user of the speech recognition system hadbeen performing prior to speaking the unrecognized word. These actionsprovide context information which may assist the speech recognitionsystem in assigning meaning to the unrecognized word.

In another embodiment of the invention, the speech recognition systemwould automatically poll the user of the speech recognition system toenter by keystrokes the information associated with the unrecognizedcommand. Assume the user spoke the same sequence as in the precedingexample, “Call Morita-san”, and the system did not recognize either thegrammatical construct “Call ______” or the name “Morita-san”. In thisembodiment of the invention, the voice-recognition system would ask theuser to illustrate the command by keystrokes and provide the name bykeystrokes. Accordingly, after entry of the illustrative example, thespeech recognition system would then recognize that the “Call ______”construct corresponds to an instruction to perform a telephone call. Inaddition, after entry of the name “Morita-san” (and possibly anassociated telephone number), the speech recognition system wouldrecognize “Morita-san” as someone to be called at a specific telephonenumber.

Further embodiments of the present invention implement additionalfeatures that may be used in combination with the functionalityassociated with the foregoing aspects of the present invention. Forexample, often a user of a speech recognition system provides biometriccues identifying when the user is introducing a new word. The user mayslow down her speech to emphasize a word, may speak more loudly toemphasize a word, or may pause to emphasize a word. These actions may beused alone or in combination with physical gestures to emphasize a word.Further embodiments of the present invention employ audio and visualbiometric measuring systems to help identify when a user of a speechrecognition system is speaking a new word.

Now further aspects of the present invention, and the problems theyovercome, will be described in greater detail. There are two typicalsituations encountered in a speech recognition system with respect tonew words. In a first situation, the speech recognition systemrecognizes a word as a valid phonetic sequence known to at least oneacoustic vocabulary of the speech recognition system. However, the wordis used in new way not recognized by an embedded grammar of a languagemodel incorporated in the speech recognition system. “Embedded grammar”and “language model” are concepts and means for implementing a speechrecognition system that generally refer to the fact that a speechrecognition system recognizes and assigns meaning to not only words, butto combinations of words. In a voice-activated user interfaceincorporating a speech recognition system, “embedded grammar” and“language model” refer to the functionality of the speech recognitionsystem that recognizes both responses to queries initiated by thevoice-activated user interface, and to commands entered by a user of thevoice-activated user interface. So in the first example, a word that isrecognized as a valid phonetic sequence is nonetheless used in a such away that the speech recognition system cannot assign meaning to theutterance incorporating the word, since the word is used in a new way. Atypical example would be encountered when a word that is recognized by avoice-activated user interface as a valid phonetic sequence is used in acommand, wherein the embedded grammar functionality which ordinarilydetects the command is not programmed to recognize and assign meaning tothe command when the command incorporates the new word. In one aspect ofthe present invention various methods and apparatus are provided thatenable an embedded grammar of a speech recognition system to “grow” byadding new words to the embedded grammar.

In a more general situation, a sequence of sounds corresponding to oneor more words spoken by a user of a speech recognition system may beunknown to any vocabulary or language model of the speech recognitionsystem. In this aspect of the present invention, various methods andapparatus are provided that enable a speech recognition system to growboth by adding previously unknown words to one or more vocabularies ofthe speech recognition system, and by adding new grammatical constructs(such as, for example, new commands) to an embedded grammar of alanguage model incorporated in a speech recognition system.

Embodiments of the present invention responding to the firstcircumstance identified above—where a known word is used in a new,unrecognized context—are handled in the following manner. Generally, anembedded grammar incorporated in a language model of a speechrecognition system operating in accordance with the invention isdesigned to expand by accommodating new uses for words recognized byother aspects of the speech recognition system (such as phoneticvocabularies).

A conventional embedded grammar operates as follows when a word includedin the grammar is spoken:

-   -   Construct: {W1} {W2}        -   Prepare list of acceptable Li's        -   L1, L2, . . . are all list items—part of an embedded grammar        -   L1, . . . Ln are all equi-probable (to a first degree of            approximation)        -   For example, Call <name>, where name may be a list of 50            proper names        -   Phrase score for {W1} {W2} {Li}        -   =Acoustic score (Li)+Language Model Score (Li|W1W2)            As is apparent, a particular word Li having the highest sum            for acoustic score and language model score is deemed to be            the most likely hypothesis for the word intended by a            speaker. No accommodation is made in conventional methods            for words unrecognized by the speech recognition system.

In contrast, in methods and apparatus of the present invention, embeddedgrammars and language models of a speech recognition can expand toincorporate words that are recognized by other aspects of the speechrecognition system (such as, for example a phonetic vocabulary), butwhich are not recognized by a particular embedded grammar as a validoption. A method of the present invention operates in the followingmanner:

-   -   (‘U’ (Word actually spoken) is not in an embedded grammar)        Construct: {W1} {W2}        -   “Create” an empty list item and assign it a non-zero            probability, P{U}<P{Li}        -   Word (‘U’) recognized by other aspects of speech recognition            system but not by embedded grammar has a small probability            allowing grammar room to expand        -   For example, “Go to <city not in embedded grammar>        -   P{U}<P{Li}, but Acoustic Score (U)+Language Model        -   Score (U)>Acoustic Model Score (Li)+Language Model Score            (Li)            In this method of the present invention, the sum of the            acoustic and language model scores will favor the word            recognized by other aspects of the speech recognition system            (such as a phonetic vocabulary) but not by the embedded            grammar over words that are recognized by the embedded            grammar. This results from the fact that none of the words            initially in the embedded grammar sound like the word            actually spoken. Alternatively, the word not in the embedded            grammar is recognized phonetically with a high degree of            probability since the word is in at least one phonetic            vocabulary of the speech recognition system. Accordingly,            the speech recognition system concludes that the most likely            hypothesis is that the speaker intended to use the new word            in, for example, the command spoken, as opposed to any words            recognized by the embedded grammar.

A method operating in accordance with this aspect of the presentinvention may be followed by additional steps. For example, the speechrecognition system may synthesize a hypothesis corresponding to theutterance spoken by the speaker and play it to the speaker using theword not initially in the embedded grammar but incorporated in someother vocabulary or grammar of the speech recognition system. In such aninstance the system would seek confirmation from the speaker that theword is what the speaker intended. As part of these additional steps, abaseform may be generated so that pronunciation can be confirmed.

In the other situation described above where an utterance isunrecognized by any vocabulary or grammar of a speech recognitionsystem, the present invention operates on phone sequences to generatehypotheses for a word or combinations of words spoken by a user that areunrecognized by the speech recognition system. A speech recognitionsystem operating in accordance with the present invention generates ahypothesis and assigns a confidence score to check if a hypotheticalword corresponds to the spoken word with a high degree of probability.The speech recognition system can seek confirmation from a speaker tomake sure the system reproduced the correct word. For example, if thespeaker spoke the command “Call Moscow” and the word “Moscow” is not inany vocabulary or grammar of the speech recognition system, the speechrecognition system would reproduce the sound sequence “moss cow” andcompute a confidence score for the combination of syllables. This aspectof the present invention operates based on the assumption that it ispossible to understand what a user spoke by identifying sequences ofsyllables. In order for the speech recognition system to implement thisaspect of the present invention, the system incorporates a library thatincludes possible phones or syllables that might occur in a user'sactive vocabulary. In addition, the system includes decoding graphsindicating how individual phones or syllables can be combined.

In a typical implementation, this second aspect of the present inventionwould operate in combination with the first aspect. For example, in manyinstances, it would not be necessary for the system to operate withphone or syllable decoding enabled at all times, since the user would bespeaking words that are recognized at least by phonetic vocabularies ofthe speech recognition system. However, when an utterance is encounteredwhich is not recognized by any vocabulary or grammar of the speechrecognition system, the phone/syllable decoder of the present inventionwould be enabled to assist in decoding of the utterance.

Various embodiments of the invention operate to improve the efficiencyof a speech recognition system in identifying new words based onphonetic methods. For example, in one embodiment a database of viablephone/syllable sequences and associated combination probabilities isimplemented to assist the speech recognition system in proposing word orutterance hypotheses with a high degree of confidence. The combinationprobabilities may reflect the likelihood of a two-phone or -syllablesequence, a three-phone or -syllable sequence, etc. The viablephone/syllable sequence database can be implemented in many ways inaccordance with the present invention. For example, the viablephone/syllable sequence database can reflect phone/syllable sequenceslikely to be encountered in interactions with a particular user of aspeech recognition system; phone/syllable sequences likely to beencountered with respect to a set of commands used in combination with avoice-activated user interface; phone/syllable sequences likely to beencountered in proper names and surnames; phone/syllable sequenceslikely to be encountered in a specific language; and phone/syllablesequences likely to be encountered in a subset of languages or alllanguages.

In further embodiments of the invention additional information—such as,for example speech and body movement biometric information—are used toidentify new words. Apparatus associated with the speech recognitionsystem detect changes in speech cadence which may be indicative of a newword. Additional apparatus associated with the speech recognition systemanalyze video data to detect gestures and body movements that may beindicative of introduction of a new word in the speech of a user of aspeech recognition system.

FIG. 1 is a block diagram showing a plurality of systems thatselectively may be incorporated in various embodiments of the presentinvention. The central system is an hierarchical mapping system 100 thatreceives inputs from a plurality of interconnected systems comprisingthe speech recognition system 10 of the present invention. Thehierarchical mapping system 100 processes information received fromother blocks and maps user input (such as, for example, a voiceutterance) into a vocabulary subset in a hierarchical open set 105 ofvocabularies. In an example, the hierarchical mapping process 100 candecode an acoustic utterance “China” into a word “China” that may belongto one of the system vocabularies but which is not recognized by agrammar set associated with a context in which the word “China”appeared. In the speech recognition system of the present invention, thehierarchical mapping process 100 adds “China” to the grammar setassociated with the context in which the word “China” appeared andinterprets the utterance in accordance (via semantic/context interpreter120) with the context otherwise indicated by the utterance. A particularadvantage of the present invention results from the fact that openhierarchical set 105 is comprised of open subsets (grammars andvocabularies)—as a result, these subsets are dynamic and can be updatedwith new words and grammatic constructs in various embodiments of thepresent invention. Learning module 103 is operable to learn userbehavior associated with user requests (using internet facilities tolearn across a plurality of users) and associate commands to userrequests. In one example, a previously unrecognized command like “CallChina” would be associated with an action to call a specific telephonenumber after the speech recognition system learns the word “China” andthrough interaction with a user learns to associate the command “CallChina” with the action to call a specific telephone number.

Confidence score metrics system 104 resolves conflicts between differentwords and their membership in different subsets in the hierarchy. Forexample, referring back to the “Call China” example, there may be a wordincorporated in a grammar which has a higher language model score than“China” but which has a lower acoustic score than “China”. Theconfidence score metrics system 104 operates in such a way to resolvethese conflicts. In various embodiments of the invention, confidencescores can be assigned for acoustic models, language models and forsemantic models. In embodiments of the present invention an acousticscore is assigned for a sequence of phones or syllables viaphone/syllable mapper 102. The acoustic representation determined with ahigh degree of confidence from this scoring process may not correspondto any existing word in a set of vocabularies 106, 107, 108 or 109. Insuch a situation, if the confidence score block 104 evaluates theconfidence metric for a new phone/syllable sequence as higher than thescore for competitive words—the new sequence of phones/syllables will beconsidered as a new word that should be added to an open vocabulary(e.g., to 109). A meaning for the new word/phrase is received throughone or both of user actions learning module 103 and semantics/contextinterpreter block 120. New commands are also added to a grammar 106 inembodiments of the present invention. Language model services block 107provides language data for sequences: phones, syllables, words andphrases. This data can be used by the confidence score block 104 toderive confidence scores. This language data also can be used to computelanguage model scores in a decoding process operating within thehierarchical mapping system 100. User behavior biometric detector 101provides biometrics data about user behavior (e.g., conversationalbiometrics) that helps to identify whether the acoustic utterance pointsto a new word (e.g., hesitation on some phrases, pauses, speaking stressetc.).

FIG. 2 is a block diagram depicting in greater detail the hierarchicalmapping system 100 of FIG. 1. The hierarchical mapping system 100contains a communications bus 200 through which different system modulesexchange data. Data that enters hierarchical map system 100 (through bus200) comprises data produced by modules previously described withrespect to FIG. 1 and which are connected to 100 (e.g., speech inputfrom 110, phonetic data in 203 from 102, confidence data in 203 from104, biometrics data in 203 from 101 etc.).

Speech input 201 is directed to the hierarchical speech recognitionsystem 202. This speech system operates to provide hierarchical decodingof, for example, phones, syllables, words and phrases. Hierarchicalspeech recognition system 202 also produces data for computation ofhierarchical scores in 204.

Hierarchical score calculator 204 also uses conventional biometricsinformation from user biometric detector 101. For example, if the userhesitates on some acoustic utterance—a score is added to the confidencescore for acoustic information (for example as linear weighted sum). Forexample, duration of hesitation or stress value of sounds may benormalized and added as a weighted sum. Similarly other scores(semantic, language models etc) are added as a weighted sum in morecomplex implementations. The confidence score is computed either forseparate words, for phonetic/syllable sequences, or for membership insome subset (a grammar, vocabulary etc.) in 205. If a novel sequence ofphones/syllables/phrases is chosen (via the highest confidence score) itis added by the vocabulary extender 206 to the appropriate subset.

FIG. 3 depicts a phone/syllable mapper 102 capable of operating inaccordance with the present invention. Phonetic decoder 300 is aphonetic/syllable decoder that can be used in an hierarchical speechrecognition system 202 that decodes phonetically. The phone/syllabledecoder 300 uses phone 306 and syllable vocabularies 309 andphone/syllable language models 308 for decoding. Phone 306 and syllablevocabularies 309 are created from a database of names 301 (which caninclude names in different languages—English 302; Chinese 303; Japanese304; Indian dialects 305). Other databases for other categories includephone classes 312 and a universal phonetic system 311 (that areapplicable to several or all languages). Language models 308 andphone/syllable vocabularies 306, 309 are used to create viablephonetic/syllable sequences 310, which are derived from viable languagemodels stored in a database or which are created dynamically. Theseviable sequences are not words in some vocabulary, but have a goodchance to become legitimate words to be added to open vocabularies.

FIG. 4 depicts a user behavioral biometrics detector 101. Userbehavioral biometrics detector 101 comprises a speech input 400 and avideo input 401. Pause detector 402 operates to detect pauses in speech;stress volume indicator 403 operates to detect stresses during speech;and speech speed measure detector 404 operates to detect changes inspeech speed. Speech biometrics interpreter 408 combines informationderived from the speech data by 402, 403 and 404.

Video data received at input 401 is operated on by head positiondetector 405, body movement detector 406, and gesture detector 407. Headposition detector 405 that helps to identify whether a user requestedsome actions from a system by looking at a device—for example, bylooking at a window in a car and asking to open the window. Informationderived by 405, 406 and 407 are combined by body movements/gestureinterpreter 409 to provide a complete biometrics picture based on usermovement

FIG. 5 is a flow chart depicting a method of the present invention. Atstep 500, a speech recognition system capable of practicing the methodsof the present invention receives an utterance and process theutterance. Then, at step 501, the speech recognition system decodes theacoustic data. Step 502 is a decision point where the speech recognitionsystem decides whether the entire acoustic utterance has been decoded.If it has, then at step 503 the speech recognition system interprets theacoustic data. At step 506, the speech recognition system reachesanother decision point. At step 506, the speech recognition systemdecides whether the entire utterance has been interpreted. If so, atstep 507, a command contained in the utterance is executed.

Returning to step 502, if the entire acoustic utterance cannot bedecoded, the speech recognition system decides whether the utterance canbe decoded in an extended system. If so, it continues to step 506. Ifthe entire utterance cannot be decoded in the extended system, thesystem continues to step 505 which is another decision point. At step505, the speech recognition system determines whether there isadditional biometric/context data available that points to a new word.If so, the speech recognition systems continues to step 520, where userbiometric data is interpreted either implicitly or by asking questions.Then at step 509 the vocabulary is updated. If not, the utterance usinterpreted by interacting with the user.

One of ordinary skill in the art will understand that the methodsdepicted and described herein can be embodied in a tangiblemachine-readable memory medium. A computer program fixed in a machinereadable memory medium and embodying a method or methods of the presentinvention performs steps of the method or methods when executed by adigital processing apparatus coupled to the machine-readable memorymedium. Tangible machine-readable memory media include, but are notlimited to, hard drives, CD- or DVD-ROM, flash memory storage devices orin a RAM memory of a computer system.

Thus it is seen that the foregoing description has provided by way ofexemplary and non-limiting examples a full and informative descriptionof the best method and apparatus presently contemplated by the inventorsfor implementing a speech recognition system for identifying, andassigning meaning to, new words and utterances initially unknown to thespeech recognition system. One skilled in the art will appreciate thatthe various embodiments described herein can be practiced individually;in combination with one or more other embodiments described herein; orin combination with speech recognition systems differing from thosedescribed herein. Further, one skilled in the art will appreciate thatthe present invention can be practiced by other than the describedembodiments; that these described embodiments are presented for thepurposes of illustration and not of limitation; and that the presentinvention is therefore limited only by the claims which follow.

1. A signal-bearing medium tangibly embodying a program ofmachine-readable instructions executable by a digital processingapparatus of a computer system to perform speech recognition operations,the speech recognition operations comprising: detecting at least atarget word known to an acoustic vocabulary but unknown to an embeddedgrammar of a language model of the speech recognition system; assigninga language model probability to the target word; calculating a sum of anacoustic and language model confidence score for the target word andwords already included in the embedded grammar of the language model;and if the sum of the acoustic and language model probability for thetarget word is greater than the sum of the acoustic and language modelprobability for the words already included in the embedded grammar,adding the target word to the language model.
 2. The signal-bearingmedium of claim 1 where the operations further comprise: aftercalculating the sum and prior to adding the target word to the embeddedgrammar of the language model, asking confirmation of the target wordfrom a user of the speech recognition system; and receiving confirmationfor the target word from the user of the speech recognition system. 3.The signal-bearing medium of claim 2 wherein confirmation comprisesconfirmation of the spelling of the target word.
 4. The signal-bearingmedium of claim 2 wherein confirmation comprises confirmation of thepronunciation of the target word.
 5. A signal-bearing medium tangiblyembodying a program of machine-readable instructions executable by adigital processing apparatus of a computer system to perform speechrecognition operations, the speech recognition operations comprising:detecting an utterance having a low acoustic score within an acousticvocabulary of the speech recognition system indicating that theutterance may correspond to an out-of-vocabulary word; generating atleast one new word hypothesis comprised of at least one of a phone- orsyllable sequence using confidence scores derived from probabilitiescontained in a database of viable phone and syllable sequences; and ifthe at least one new word hypothesis meets a pre-determined criterion,adding a word corresponding to the at least one new word hypothesis tothe vocabulary of the speech recognition system.
 6. The signal-bearingmedium of claim 5 wherein the pre-determined criterion corresponds toconfirmation by a user of the speech recognition system wherein theoperations further comprise: prior to adding at least one word to theacoustic vocabulary of the speech recognition system, presenting the newword hypothesis to a user of the speech recognition system seekingconfirmation that the new word hypothesis corresponds to at least oneword intended by the user when the user spoke; and whereby the new wordis added to the vocabulary of the speech recognition system only ifconfirmation is receiving from the user.
 7. The signal-bearing medium ofclaim 6 wherein the utterance corresponds to a multi-word command, andwherein the operations further comprise: adding the command to anembedded grammar of a language model associated with the speechrecognition system.
 8. The signal-bearing medium of claim 7 wherein theoperations further comprise: adding information received from a user ofthe speech recognition system to memory indicating at least one actionto be performed when the command is detected by the speech recognitionsystem.
 9. A signal-bearing medium tangibly embodying a program ofmachine-readable instructions executable by a digital processingapparatus of a computer system to perform speech recognition operationsin a speech recognition system, the speech recognition operationscomprising: detecting an utterance not recognized by at least a firstone of an acoustic vocabulary, embedded grammar, and viablephone/syllable sequence library of the speech recognition system;generating at least one hypothesis for the utterance, wherein thehypothesis is based on information derived from a second one of anacoustic vocabulary, embedded grammar and viable phone/syllable sequencelibrary of the speech recognition system; calculating a confidence scorefor the at least one hypothesis and for members of the first one of theacoustic vocabulary, embedded grammar and viable phone/syllable sequencelibrary of the speech recognition system; comparing the confidencescores calculated for the at least one hypothesis and for members of thefirst one of the acoustic vocabulary, embedded grammar and viablephone/syllable sequence library of the speech recognition system; andadding information to the first one of an acoustic vocabulary, embeddedgrammar and viable phone/syllable sequence corresponding to thehypothesis if a pre-determined criterion based on the comparison is met.10. The signal-bearing medium of claim 9 wherein the utterancecorresponds to a phone sequence, and wherein the first one of theacoustic vocabulary, embedded grammar and viable phone/syllable sequencelibrary corresponds to a particular viable phone/syllable sequencelibrary.
 11. The signal-bearing medium of claim 9 wherein the utterancecorresponds to a word, and wherein the first one of the acousticvocabulary, embedded grammar and viable phone/syllable sequence librarycorresponds to a particular acoustic vocabulary.
 12. The signal-bearingmedium of claim 9 wherein the utterance corresponds to a command, andwherein the first one of the acoustic vocabulary, embedded grammar andviable phone/syllable sequence library corresponds to a particularembedded grammar.
 13. The signal-bearing medium of claim 9 wherein theat least one criterion corresponds to confirmation by a user of thespeech recognition system, wherein the operations further comprise:prior to adding information corresponding to the at least one hypothesisto the first one of the acoustic vocabulary, embedded grammar and viablephone/syllable sequence library of the speech recognition system,seeking confirmation that the hypothesis corresponds to what the userintended when the user spoke; and whereby the information is added onlyif confirmation is received from the user of the speech recognitionsystem.
 14. The signal-bearing medium of claim 9 wherein the operationsfurther comprise: using biometric information to assist in identifyingthe utterance as unrecognized by the first one of the acousticvocabulary, embedded grammar and viable phone/syllable sequence libraryof the speech recognition system.
 15. The signal signal-bearing mediumof claim 14 wherein the biometric information comprises speech biometricinformation.
 16. The signal-bearing medium of claim 14 wherein thebiometric information comprises data derived from video information. 17.A speech recognition system comprising: a speech input for receivingspeech from a user of the speech recognition system; an open setcomprised of at least one open vocabulary and at least one open embeddedgrammar associated with a language model implemented in the speechrecognition system; a hierarchical mapping system for identifyingutterances not recognized by at least one of the open vocabulary andopen embedded grammar of the speech recognition system; for generatinghypotheses for the unrecognized utterances using confidence scores basedat least in part on one of viable phone/syllable sequence information,acoustic vocabulary information and grammar information; and for addinginformation corresponding to the hypotheses to at least one of the openvocabulary and embedded grammar of the speech recognition system if apre-determined criterion is met; and a confidence score system forgenerating confidence scores for use by the hierarchical mapping system.18. The speech recognition system of claim 17 further comprising: a userbehavior biometrics detector for generating data to assist thehierarchical mapping system in identifying utterances that a userexpects not to be recognized by the speech recognition system.
 19. Thespeech recognition system of claim 17 further comprising: a confirmationsystem for providing the hypotheses corresponding to the unrecognizedutterances to a user of the speech recognition system, and for receivingconfirmation from the user if the hypotheses correspond to what the userintended when the user spoke the unrecognized utterances.
 20. The speechrecognition system of claim 17 further comprising: a user input systemfor receiving data from the user of the speech recognition system,wherein the data is associated with the information corresponding to thehypotheses added to at least one of the open acoustic vocabulary andopen embedded grammar of the speech recognition system when apre-determined criterion is met.
 21. The speech recognition system ofclaim 17 wherein the data concerns at least one action to be performed.